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SUMMARY 


A  critical  linkage  in  training  systems  is  the  translation  of 
training  evaluation  information  for  reassessing  training 
needs  and  for  making  training  program  changes.  This  report 
presents  the  development  of  a  conceptual  framework  for 
examining  the  job  relevancy  and  efficiency  of  training  and 
the  linkage  of  this  evaluative  information  to  training  needs 
reassessment.  How  to  integrate  job  performance  information 
into  the  existing  training  evaluation  system  for  identifying 
over-  and  undertrained  tasks  is  also  described.  The 
Aerospace  Ground  Equipment  (AGE)  (AFS423X5)  Technical 
Training  Program  is  used  as  a  case  analysis  for  this  report. 
Recommendations  for  future  research  to  identify  the  content 
domain  of  an  Airman  Basic-in-Residence  (ABR)  Training 
program  and  to  integrate  performance  information  into  the 
evaluation  system  is  presented. 
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PREFACE 
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to  Air  Force  technical  skills  training. 
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Chanute  AFB  for  their  time  and  insight  into  the  technical 
training  system:  SMSGT  Kevin  Watt,  Mr.  John  Predmore  and 
Mr.  Gissing.  Thanks  to  Mr.  Greer  for  scheduling  our  trips 
to  Chanute  AFB.  Thanks  also  to  members  of  the  Occupational 
Measurement  Center  at  Randolph  AFB  (Col.  Baker,  Mr. 

Tartell,  Captain  Agee,  Lt.  Col.  Houtman,  Dr.  Aslett)  for 
providing  information  about  the  occupational  survey  process. 
Finally,  special  thanks  to  Dr.  Henrick  Ruck  for  his  insights 
into  the  research  issues  relevant  to  training  evaluation. 
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LINKING  TRAINING  EVALUATION  TO  TRAINING  NEEDS  ASSESSMENT: 


DEVELOPMENT  OF  A  CONCEPTUAL  MODEL 


I .  INTRODUCTION 

In  fiscal  year  1989,  over  35,000  airmen  received 
initial  technical  skills  training  at  resident  Air  Training 
Command  (ATC)  schools  to  prepare  them  for  first  job 
assignments  in  the  Air  Force.  This  represents  an  enormous 
amount  of  resources  devoted  to  initial  technical  training. 
Training  that  is  irrelevant  or  inappropriate  at  the 
technical  training  level  is  therefore  quite  costly  to  the 
Air  Force  (Ruck,  Thompson,  &  Stacy,  1987).  Active  steps 
must  be  taken  to  ensure  that  technical  training  programs  are 
job-related  and  efficient. 

The  Instructional  System  Development  Process 

Each  initial  technical  training  program  is  planned, 
developed  and  conducted  through  a  systematic  and  complex 
five  step  process.  This  Instructional  System  Development 
( ISD)  process  which  guides  the  development  and  revision  of 
technical  training  programs  includes  the  following  steps: 

(a)  analyze  system  requirements;  (b)  define  education  and 
training  requirements;  (c)  develop  objectives  and  tests;  (d) 
plan,  develop,  and  validate  instruction;  and  (e)  conduct  and 
evaluate  instruction  (AF  Manual  50-2,  1979) . 

The  fifth  step  in  the  ISD  process  specifies  that  a 
thorough  evaluation  of  the  training  program  must  be 
conducted.  The  linkage  between  the  evaluation  phase  (step 
5)  and  the  analysis  of  system  requirements  or  training  needs 
assessment  (step  1)  implies  that  training  is  a  continuously 
evolving  process  that  uses  evaluative  information  to  adapt 
and  to  improve  its  quality.  Therefore,  a  critical  linkage  in 
the  ISD  process  is  the  translation  of  training  evaluation 
information  into  input  useful  for  reassessing  training  needs 
and  for  making  meaningful  training  program  changes. 

Objective  of  Project 

As  part  of  the  ISD  process,  ATC  has  developed  a  formal 
training  evaluation  system  to  allow  for  the  orderly 
development  of  change  in  the  technical  training  programs. 
Given  that  the  evaluation  system  was  developed  over  10  years 
ago,  it  would  seem  reasonable  to  examine  and  re-evaluate  the 
comprehensiveness  and  effectiveness  of  the  evaluation 
system.  This  examination  focuses  on  whether  or  not  the 
evaluation  system  is  limited  in  its  ability  to  succinctly 
and  reliably  answer  questions  about  the  quality  of  technical 
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training  programs  or  to  provide  the  type  of  information 
required  for  use  by  training  managers  in  making  course 
revisions  and  continuous  improvements. 

To  critically  examine  the  training  evaluation  system,  a 
conceptual  framework  must  be  developed  to  guide  the  effort. 
The  objectives  of  this  project  are  to:  (a)  develop 
conceptual  frameworks  for  understanding  the  components  of  a 
comprehensive  training  evaluation  system,  (b)  describe  the 
training  evaluation  system  used  in  ATC  training  programs 
through  a  case  analysis  of  a  specific  technical  training 
program,  (c)  critically  examine  the  training  evaluation 
system  in  relation  to  the  conceptual  models  developed,  and 
(d)  present  recommendations  for  research  to  examine  the 
complex  issues  of  better  linking  training  evaluation 
information  to  training  needs  reassessment. 


II.  A  CONCEPTUAL  FRAMEWORK  FOR  TRAINING  EVALUATION 

A  comprehensive  analysis  of  training  needs  is  critical 
to  training  program  development.  Similarly,  an  initial 
assessment  of  training  needs  should  also  provide  information 
for  the  development  of  a  comprehensive  training  evaluation 
system.  The  key  question  asked  during  this  evaluation 
development  phase  is  "what  do  we  want  to  know  about  the 
training  program  that  will  indicate  the  quality  of  the 
program?"  In  this  regard,  decision  makers  may  be  interested 
in  answering  one  or  more  evaluation  questions  i.e., 
evaluation  may  be  required  for  different  purposes.  To 
answer  each  evaluation  question  or  to  fulfill  each  purpose, 
different  types  of  information  or  data  must  be  gathered  by 
an  evaluation  system.  The  conceptual  framework  relating 
training  evaluation's  purpose  and  types  of  evaluative 
information  needed  is  presented  below. 

A  Typology  of  Training  Evaluation  Purposes 

The  purpose  of  the  evaluation  drives  the  kinds  of 
information  that  needs  to  be  collected.  Thus,  if  the 
purpose (s)  behind  the  development  of  an  evaluation  system 
for  a  particular  training  program  is  not  clearly  specified 
during  the  initial  analysis  or  needs  assessment  phase,  it  is 
unlikely  that  the  appropriate  information  will  be  collected 
to  link  subsequent  evaluation  data  with  training  needs 
reassessment.  Without  a  clear  sense  of  purpose, 
modifications  to  enhance  the  quality  of  the  training  program 
must  rely  on  the  intuition  of  the  people  who  develop  the 
training  program  (Montague  &  Wulfeck,  1986) . 

There  are  five  major  purposes  for  conducting  training 
evaluation.  Table  1  presents  the  five  purposes,  the 
questions  asked  for  each  purpose  and  the  types  of 
information  needed  to  answer  the  questions  posed  by  each. 
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The  Purposes  for  Conducting  Training  Evaluation 
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Predict  Job  Pertorm.nce?  Job  Pertormence 


One  purpose  for  conducting  training  evaluation  is  to 
determine  the  content  validity  of  the  training  program 
(Goldstein,  1986) .  Content  validity  asks  the  question  "is 
the  training  content  job  relevant?."  To  answer  this 
question,  information  about  the  tasks  being  performed  on  the 
job  must  be  matched  with  the  tasks  being  trained  in  the 
training  program.  A  content  validity  ratio  can  be 
calculated  to  determine  the  extent  to  which  the  content  of 
the  training  program  is  job  relevant  (Ford  &  Wroten,  1984)  . 

A  second  training  evaluation  purpose  is  to  examine 
training  efficiency.  Training  efficiency  asks  the  question 
"is  the  training  program  over-  or  undertraining  certain 
tasks  (or  Knowledges,  Skills  and  Abilities  (KSAs))?."  To 
answer  this  question,  information  regarding  the  importance 
of  tasks  in  the  job  domain  and  the  emphasis  of  those  tasks 
in  the  training  program  must  be  acquired.  A  matching 
technique  can  then  be  applied  to  examine  areas  of  over-  or 
undertraining  (Ford  &  Wroten,  1984)  .  Ore  outcome  of  this 
analysis  could  be  the  elimination  of  cercain  tasks  from  the 
technical  school  training  program  that  might  be  better 
trained  on  the  job. 

A  third  purpose  for  conducting  training  evaluation  is 
to  determine  training  validity.  Goldstein  (1986)  states 
that  this  purpose  asks  the  question  "did  the  trainees  learn 
the  material  that  was  being  trained?."  To  answer  this 
evaluation  question,  one  must  collect  information  about  the 
performance  of  the  trainees  during  and/or  at  the  end  of  the 
training  program.  The  analysis  determines  the  extent  to 
which  learning  has  taken  place  in  comparison  to  a  specified 
standard  or  criterion  of  success. 

A  fourth  purpose  of  evaluation  is  to  determine  transfer 
validity  (Wexley,  1984) .  Transfer  validity  asks  the 
question  "are  people  performing  well  on  the  job  after 
training?."  To  answer  this  question,  information  regarding 
performance  on  the  job  must  be  collected.  Job  performance 
must  then  be  compared  to  some  criterion  of  success  to 
determine  the  extent  of  successful  transfer.  Experimental 
designs  can  be  used  to  determine  if  the  transfer  of 
knowledges  and  skills  have  actually  occurred  and  whether  the 
change  is  attributable  to  the  training  program. 

A  fifth  purpose  of  training  evaluation  is  to  determine 
the  predictive  validity  of  a  training  program.  At  times, 
training  is  used  as  a  device  to  select  or  place  individuals 
into  a  particular  position  or  job.  Therefore,  the  question 
that  must  be  answered  with  this  evaluation  purpose  is  "does 
training  performance  predict  job  performance  so  that 
accurate  selection/placement  decisions  can  be  made?."  The 
information  that  must  be  collected  to  conduct  this  analysis 
includes  training  and  job  performance  information. 
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Types  of  Training  Evaluation  Information 


Two  major  dimensions  of  information  available  for 
answering  the  questions  posited  for  each  evaluation  purpose 
are  represented  in  Table  1.  One  dimension  is  the  source  of 
the  information  that  is  gathered.  The  two  sources  of 
information  available  for  a  comprehensive  training 
evaluation  system  ire  from  the  job  domain  and  the  training 
domain.  A  second  dimension  is  the  type  of  information 
gathered.  The  type  of  information  is  conceptualized  as 
being  task-based  or  performance-based.  Task-based 
information  focuses  on  what  an  individual  "does  do"  while 
performance  information  examines  "how  well  an  individual 
does  the  specified  tasks." 

Figure  1  presents  a  conceptual  representation  of  the 
four  domains  that  result  from  combining  the  two  types  of 
sources  with  the  two  types  of  information  available.  The 
four  domains  are  the:  (a)  job  content  domain,  (b)  training 
content  domain,  (c)  job  performance  domain,  and  the  (d) 
training  performance  domain. 

The  job  and  training  content  domains  are  conceptualized 
as  consisting  of  a  task  and  an  emphasis  component.  The 
"task"  component  of  the  job  content  domain  involves  the 
identification  of  what  tasks  (or  KSAs)  are  performed  on  the 
job.  The  "task"  component  of  the  training  content  domain 
includes  information  regarding  what  tasks  (or  KSAs)  are 
taught  during  training. 

The  "emphasis"  component  of  the  job  content  domain 
involves  an  analysis  of  how  "important"  the  various  tasks 
t  ~e  for  job  performance.  Importance  can  be  operationalized 
in  a  number  of  ways  including  "importance  to  job 
performance"  and  "task  difficulty."  The  "emphasis" 
component  of  the  training  content  domain  concerns  the  amount 
of  effort  devoted  to  training  the  various  tasks  included  in 
the  training  program.  Emphasis  can  be  operationalized  as  a 
subjective  assessment  of  effort  devoted  in  training  to  each 
task  or  as  the  actual  amount  of  time  spent  training  each 
task. 


The  job  and  training  performance  domains  are 
conceptualized  as  consisting  of  a  knowledge  and  a 
performance  component.  The  "knowledge"  component  of  the  job 
performance  domain  focuses  on  how  well  the  individual  knows 
the  appropriate  ways  to  perform  job  tasks.  Job  knowledge 
tests  are  often  used  to  address  this  issue.  The 
"performance"  component  specifies  how  well  the  individual 
actually  performs  the  job  tasks.  Information  from  job 
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performance  ratings,  work  samples  and  job  indices  such  as 
quantity  or  quality  of  work  can  be  gathered  to  determine  the 
level  of  job  performance. 

For  the  training  performance  domain,  the  "knowledge" 
component  examines  how  well  the  trainees  have  learned  the 
material  that  has  been  taught.  Paper-and-pencil  tests  on 
the  material  covered  in  the  course  are  typically  used  to 
examine  the  knowledge  component.  The  "performance" 
component  focuses  on  how  well  the  trainees  can  perform  the 
tasks  that  have  been  trained.  Performance  ratings,  work 
samples,  and  other  objective  indices  can  be  used  to 
determine  performance  level  while  the  trainees  are 
progressing  through  the  training  program. 

Summary 

In  summary,  the  model  in  Figure  1  provides  a  conceptual 
framework  for  describing  the  information  available  to  answer 
training  evaluation  questions.  The  specific  information 
gathered  is  dependent  upon  the  purpose (s)  for  conducting 
training  evaluation  (see  Table  1) .  An  examination  of  Figure 
1  also  indicates  that  a  critical  boundary  condition 
surrounding  these  four  domains  is  information  quality.  This 
means  that  the  usefulness  of  any  information  collected  for 
answering  the  evaluation  question  posed  is  heavily  dependent 
upon  the  quality  of  that  information. 

III.  ANALYSIS  OF  THE  AF  ABR  TRAINING  EVALUATION  SYSTEM 

The  conceptual  models  developed  in  this  paper  highlight 
the  important  role  evaluation  systems  have  for  enhancing  the 
quality  of  training.  This  section  describes  the  existing 
evaluation  system  relevant  to  the  four  domains  of 
information  presented  in  Figure  1.  The  Aerospace  Ground 
Equipment  (AGE)  (AFS423X5)  Airmen  Basic-in-Residence  (ABR) 
course  at  Chanute  AFB,  Illinois  is  used  as  a  case  study. 

Training  Evaluation  Information 

There  are  many  documents  in  the  Air  Force  that  provide 
information  that  is  useful  in  defining  the  job  and  training 
content  and  performance  domains  (see  Appendix  A  for 
listing) .  Table  2  lists  the  four  domains  along  with  the 
sources  of  information  available  in  the  Air  Force  training 
evaluation  system,  the  type  of  information  they  contain,  and 
the  level  of  specificity  of  the  various  types  of  evaluative 
information. 

Level  of  specificity  refers  to  the  amount  of  detail  in 
the  evaluative  information  collected.  The  Air  Force  uses  a 
task-  based  approach  to  needs  assessment  and  training 
evaluation.  An  examination  of  this  evaluation  system  reveals 
three  levels  of  specificity  in  the  task-based  data  that  is 
collected.  The  basic  component  of  the  evaluation  system  is 
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Table  2.  The 

Available  Sources  of  Evaluation 

Information. 

Source  of 

Type  of 

Level  of 

Domain 

Information 

Information  Specificity 

Job  Content 

OSR 

Task 

T 

Emphasis 

T 

STS 

Task 

CT 

Knowledge 

CT 

U&TW 

Task 

CT  &  T 

Training 

Content 

POI 

Task 

CT 

Job 

QA  Checks 

Performance 

T  &  TOSL 

Performance 

TER 

Task 

CT 

Performance 

CT 

TQR 

Task 

CT 

Performance 

CT 

Training 

Block 

Performance 

Tests 

Knowledge 

TOSL 

Performance 

Progress 

Checks 

Performance 

T 

T  =>  Task,  CT  =>  Cluster  of  Tasks,  TOSL  =>  Technical  Order 
Step 
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at  the  "task"  (labeled  T  in  Table  2)  level.  A  task  is 
defined  as  a  unit  of  work  activity  or  operation  which  forms 
a  significant  part  of  a  duty  (AF  Manual  50-  2,  1979).  An 
example  of  task  level  information  is  found  in  the 
Occupational  Survey  Report  (OSR) . 

An  OSR,  a  product  of  the  Occupational  Measurement  Center 
(OMC) ,  defines  the  job  content  of  a  career  field  and  is 
based  upon  a  comprehensive  survey  of  airmen  within  a  target 
occupation.  The  job  inventory  asks  the  respondents  to 
indicate  the  tasks  they  currently  perform.  In  the  OSR,  this 
information  is  recorded  in  one  of  three  time-in-service 
categories,  first  enlistment  (1-48  months) ,  second 
enlistment  (49-96  months) ,  and  career  personnel  (97+ 
months) .  Tasks  are  analyzed  to  reflect  the  percentage  of 
airmen  performing  each  task  in  each  category.  These  data 
provide  task  information  about  the  job  content  domain  by 
time-in-service.  Additional  data  collected  include  the 
relative  time  spent  performing  each  task,  the  difficulty  of 
learning  each  task  and  the  emphasis  that  should  be  placed  on 
training  each  task. 

Tasks  can  also  be  grouped  together  in  terms  of  co¬ 
performance.  An  example  of  tasks  grouped  into  more  general 
statements  is  found  in  the  Specialty  Training  Standards 
(STS)  (AF  Regulation  8-13  &  ATC  Supplement,  1,  1987)  that 
incorporate  multiple  OSR  task  statements  into  each  standard. 
We  have  labeled  this  grouping  as  the  "cluster  of  tasks"  (CT) 
level  in  Table  2 . 

Frequently,  tasks  are  broken  down  into  requisite  steps 
needed  to  perform  a  given  task.  An  example  is  the  steps 
found  in  Technical  Orders  (TO)  that  are  the  individual 
procedures  required  to  perform  each  OSR  task.  The  TOs 
provide  step  by  step  instructions  for  conducting  all  Air 
Force  equipment  maintenance.  This  includes  maintenance 
provided  for  aircraft,  electronic  systems,  support  vehicles, 
and  aerospace  ground  support  equipment.  In  Table  2,  this 
level  of  task  specificity  is  labeled  as  Technical  Order  Step 
Level  (TOSL) . 

Another  way  to  examine  the  subcomponents  of  a  task  or 
cluster  of  tasks  is  to  identify  their  underlying  job 
elements  (Primoff ,  1975) .  This  approach  specifies  the 
knowledge,  skills  and/or  abilities  (KSAs)  needed  to  perform 
each  task  or  cluster  of  tasks.  An  example  of  a  job  element 
approach  for  the  cluster  of  tasks  level  is  found  in  the  STS. 
The  STS  not  only  lists  tasks  clustered  by  co-performance, 
but  also  identifies  the  knowledge  required  to  perform  duties 
at  the  3-,  5-,  and  7-levels  of  proficiency.  An  example  of  a 
job  element  approach  for  the  task  level  of  specificity  is 
found  in  the  Training  Requirements  Analysis  (TRA)  which 
accompanies  each  new  OSR.  The  TRA,  like  the  OSR,  is  a 
product  of  OMC.  In  the  TRA,  the  underlying  KSAs  needed  to 
perform  each  OSR  task  are  identified.  This  provides 
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Training  Development  Branch  personnel  (TDB) ,  who  are 
responsible  for  designing  technical  training  courses,  with 
more  specific  information  needed  to  refine  course  content. 

The  following  is  a  detailed  description  of  the  level  of 
specificity  for  the  evaluative  information  gathered  for  each 
of  the  four  domains  described  in  Figure  1.  While  the 
information  collected  is  similar  across  career  fields,  the 
AGE  career  field  is  used  to  illustrate  the  issues  relevant 
to  training  evaluation. 

The  Aerospace  Ground  Equipment  Career  Field 

The  AGE  career  field  maintains  both  powered  and  non- 
powered  equipment  used  to  support  aircraft  in  the  Air  Force. 
The  equipment  varies  from  small  jet  engines  used  to  start 
jet  aircraft,  to  tow  bars  used  by  ground  personnel  when 
moving  aircraft.  The  ABR  course  for  AGE  lasts  18  weeks. 
After  completing  this  course,  airmen  are  sent  to  an 
operational  base  and  support  a  wide  diversity  of  Air  Force 
missions. 

Consequently,  the  ABR  course  is  required  to  teach  a 
variety  of  different  tasks  that  any  one  airman  might 
encounter  in  his  or  her  first  job  assignment. 

Job  Content  Domain 


The  most  recent  OSR  for  AGE  surveyed  2,629  airmen  in 
1983.  About  75  senior  AGE  Noncommissioned  Officers  (NCOs) 
from  different  Major  Commands  (MAJCOM)  completed  two 
additional  surveys.  The  purpose  of  these  latter  surveys  was 
to  gather  "task  difficulty"  and  "training  emphasis"  ratings 
information.  Task  difficulty  is  defined  as  the  relative 
difficulty  to  learn  a  task  and  is  rated  on  a  nine  point 
scale  from  1  (extremely  low)  to  9  (extremely  high) . 

Relative  difficulty  represents  the  time  needed,  on  the 
average,  to  learn  to  perform  a  task  satisfactorily. 

Training  emphasis  is  defined  as  the  perceptions  of  the  NCOs 
as  to  which  tasks  should  be  taught  to  first  enlistment 
airmen  in  a  structured  training  course.  This  scale  ranges 
from  0  (no  training  recommended)  to  9  (extremely  high 
training  emphasis) .  In  relation  to  the  conceptual  framework 
presented  in  Figure  1,  these  two  pieces  of  data  are  part  of 
the  emphasis  component  of  the  job  content  domain. 

In  addition  to  the  OSR  data.  Specialty  Training 
Standards  (STS)  are  created  (current  STS  423X5,  dated  Aug 
1987;  Appendix  B  provides  a  portion  of  the  current  STS  for 
AGE) .  The  1983  OSR  had  over  600  task  items  that  defined 
the  AGE  career  field.  The  AGE  STS  dated  July  1982  consisted 
of  124  statements,  of  which  80  were  statements  at  the  CT 
level  of  specificity.  The  remaining  44  STS  statements  were 
general  knowledge  statements. 
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Finally,  the  job  content  domain  also  includes  Technical 
Orders  for  the  AGE  career  field  that  are  used  whenever  a 
task  is  performed.  The  TOs  include  the  equipment  needed  as 
well  as  all  of  the  steps  needed  to  correctly  perform  the 
tasks  relevant  to  the  AGE  career  field. 

Training  Content  Domain 

The  job  content  domain  is  defined  at  three  different 
levels  of  specificity;  the  CT  level  is  defined  by  the  STS; 
the  task  level  is  defined  by  the  OSR;  and  the  step  level  is 
defined  by  the  TO.  A  TRA  for  the  AGE  career  field  has  not 
yet  been  completed. 

The  training  content  domain  is  defined  by  the  Plan  of 
Instruction  (POI) .  The  POI  is  derived  from  a  complex  system 
of  translating  OSR  and  STS  information  into  training 
objectives.  This  system  is  described  below. 

The  OSR  task  information  is  the  criterion  standard  used 
when  making  decisions  about  which  tasks  to  include  in  an  ABR 
training  course.  If  the  OSR  reports  that  30%  of  the  airmen 
in  their  first  enlistment  perform  a  task  then  the  task  is 
considered  for  possible  inclusion  in  the  course.  Also  taken 
into  consideration  by  the  AGE  TDB  is  the  task  difficulty  and 
training  emphasis  ratings. 

After  an  OSR  is  completed,  personnel  from  OMC  contact 
Subject  Matter  Experts  (SMEs)  to  collect  additional 
information  to  help  define  the  training  domain.  The  SMEs  go 
through  the  OSR  tasks  one-by-one  and  determine  within  which, 
if  any,  STS  statement  the  task  belongs.  In  some  cases,  a 
number  of  OSR  tasks  are  matched  to  one  STS  statement.  For 
example,  50  OSR  tasks  fall  within  a  single  AGE  STS 
statement;  18A.  "operational  fundamentals  of  gas  turbine 
compressors."  Some  STS  statements  are  knowledge,  not  task 
statements,  and  so  do  not  match  with  OSR  tasks. 

As  a  next  step,  a  Utilization  and  Training  Workshop 
(U&TW)  is  convened  to  review  the  ABR  training  program  and  to 
make  recommendations  for  content  changes.  The  U&TW  is 
composed  of  members  from  the  MAJCOMS,  ATC,  and  TDB.  The  STS 
is  modified  to  reflect  these  recommended  changes.  This  is  a 
critical  step  as  the  STS  indicates  which  statements  are  ABR 
course  requirements  and  the  level  of  proficiency  the 
graduate  is  expected  to  obtain  as  a  result  of  the  training. 
The  STS  is  a  contract  between  the  MAJCOMs  and  the  AGE 
school.  It  sets  the  criteria  for  knowledge  and  performance 
of  the  ABR  graduates.  The  statements  are  written  in  general 
terms  (CT  level)  in  order  to  be  as  useful  to  as  many  users 
as  possible  (e.g.  AGE  STS  statement  #11.  Reciprocating 
Engines,  b.  Diesel  Engines,  (2)  inspect.). 

After  the  U&TW  is  completed,  the  TDB  examines  the  new 
STS  and  other  relevant  material  to  identify  the  STS 
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statements  that  apply  to  the  ABR  course.  OSR  tasks  that  are 
listed  under  the  applicable  STS  statement  are  reviewed  in 
the  following  manner.  If  a  task  is  performed  by  over  50%  of 
the  airmen  during  their  first  enlistment  that  task  is 
recommended  for  knowledge  and  hands  on  training.  If  the 
percentage  performing  is  between  30  -  49%  then  the  task  is 
recommended  for  knowledge  training  only.  Below  30%  tasks 
are  generally  not  taught  in  the  ABR  course.  The  task 
difficulty  and  training  emphasis  ratings  are  also 
considered.  A  task  is  not  recommended  for  training,  no 
matter  how  high  the  percent  performing  rating  is,  if  the 
task  difficulty  rating  is  statistically  one  standard 
deviation  below  the  mean.  A  task  is  recommended  for 
training  if  the  training  emphasis  rating  is  statistically 
one  standard  deviation  above  the  mean.  The  final  criteria 
is  the  AGE  schools  ability  to  teach  the  task  given  the 
constraints  the  school  faces  with  reference  to  time, 
equipment,  qualified  instructors,  and  building  space. 

SMEs  are  also  used  to  match  the  OSR  tasks  to  the  Plan 
of  Instruction  (POI)  objectives  created  by  the  AGE  Training 
Development  Branch  (TDB) .  This  is  done  to  insure  that  the 
tasks  identified  as  needing  to  be  trained  through  the  OSR 
process  are  actually  trained  in  the  ABR  course.  Once  this 
matching  of  tasks  to  objectives  is  accomplished,  this 
matching  information  is  not  retained.  Thus,  while  the  POI 
references  STS  statements  for  each  objective,  there  is  no 
information  on  which  OSR  tasks  underlying  each  STS  are 
actually  incorporated  into  the  training  content.  This  is 
important  when  changes  are  made  in  training  content  (e.g.,  a 
new  OSR  is  completed  or  a  U&TW  inititates  course  changes)  as 
there  is  no  linkage  between  OSR  task  information  and 
training  objectives. 

The  TDB  personnel  take  the  STS  and  the  TOs  in  order  to 
rewrite  training  objectives  for  the  course.  The  TDB  then 
develop  criteria  (e.g.,  similar  equipment)  for  combining  the 
objectives  into  blocks  of  instruction.  Each  block  has  one 
or  more  objectives  and  includes  the  time  spent  training  on 
the  objective,  the  STS  statement  reference,  STS  proficiency 
level,  and  the  form  of  criteria  testing,  either  written  or 
progress  check.  For  AGE  there  are  11  blocks  of  instruction 
which  are  further  combined  into  five  general  instructional 
areas. 

Training  Performance  Domain 

The  proficiency  level  of  the  trainers  is  examined 
through  paper-and-pencil  knowledge  tests  and  hands-on 
performance  tests.  In  the  AGE  ABR  course  trainees  are  given 
knowledge  tests  at  the  end  of  each  of  the  11  blocks.  The 
questions  are  written  by  the  course  instructors  and  are 
directly  related  to  the  course  objectives.  A  major  criteria 
for  determining  a  good  test  question  is  when  less  than  50% 
of  the  trainees  get  the  question  wrong.  The  average  grade 
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on  any  given  test  is  between  85%  and  90%.  Test  questions 
are  written  at  the  sub-task  level  of  specificity.  As 
questions  are  written  from  steps  in  the  TO. 

Many  objectives  in  the  ABR  course  require  a  performance 
progress  check.  The  progress  check  involves  two  or  more 
trainees  working  together  to  perform  a  task  on  AGE  equipment 
located  at  the  technical  school.  The  trainees  use  the 
relevant  TO  to  complete  the  task  and  are  observed  at  all 
times  by  an  instructor.  If  no  difficulties  are  encountered 
during  the  performance  progress  check  the  trainees  are 
checked  off  as  having  completed  the  task.  If  the  trainees 
do  not  pass,  the  instructor  explains  what  was  done  wrong  and 
trainees  are  allowed  to  perform  the  task  again. 

Job  Performance  Domain 


Training  Evaluation  Reports  (TERs)  are  produced 
annually  or  bi-annually  by  Training  Evaluation  Division 
personnel  at  each  technical  training  command.  The  TER 
examines  how  well  the  ABR  course  is  preparing  graduates  to 
perform  their  assigned  tasks  according  to  the  proficiency 
level  specified  by  the  STS.  Surveys  are  developed  to 
collect  this  information  with  questionnaire  items  taken 
directly  from  the  applicable  STS  statements. 

For  1988,  TER  surveys  were  sent  to  314  supervisors  of 
recent  AGE  graduates  asking  them  how  satisfied  they  were 
with  the  graduate's  performance  and  if  the  tasks  taught  in 
the  course  were  being  utilized.  If  less  than  80%  of  the 
graduates  perform  the  required  tasks  satisfactorily  then 
training  is  considered  inadequate.  If  less  than  30%  of  the 
supervisors  indicate  the  graduates  are  performing  a  task 
taught  in  training,  consideration  is  given  for  removal  of 
that  task  from  the  ABR  course.  A  TER  was  recently  completed 
(Feb  88)  for  the  AGE  ABR  course.  No  recommended  changes  to 
course  content  were  made  based  on  this  information. 

Supervisors  in  the  field  have  an  additional  method  of 
providing  technical  schools  with  feedback  about  training 
effectiveness  in  the  form  of  Training  Quality  Reports  (TQR) . 
TQRs  are  submitted  by  a  supervisor  and  report  a  graduate's 
lack  of  proficiency  in  one  or  more  tasks  or  the  lack  of 
knowledge  that  the  STS  states  a  graduate  of  the  AGE  ABR 
course  should  have.  The  school  is  required  to  respond  to 
all  TQRs.  Between  January  1985  and  August  1988  7  TQRs  have 
been  submitted  about  the  AGE  ABR  course. 

In  addition,  performance  checks,  similar  to  the 
progress  checks  performed  in  the  technical  school,  are 
conducted  by  the  Quality  Assurance  (QA)  branch  personnel  in 
field  settings.  On  a  regular  basis,  QA  monitors  airmen 
performing  different  tasks  to  check  on  their  proficiency  in 
those  tasks.  These  are  tasks  they  are  used  to  certify 
proficiency.  QA  personnel  insure  that  the  tasks  are 
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completed  in  accordance  with  the  procedures  laid  out  in  the 
TOs  and  that  there  are  no  major  discrepancies.  If  a  major 
discrepancy  is  noted  then  recommendations  are  made  which 
usually  include  decertification  and/or  additional  training. 
This  information  is  used  by  the  MAJCOMs  to  monitor  wing 
readiness,  but  is  not  presently  used  to  provide  performance 
feedback  information  to  training  programs. 

IV.  CRITIQUE  OF  THE  TRAINING  EVALUATION  SYSTEM 

Table  1  presented  five  purposes  for  conducting  training 
evaluation.  A  critique  of  the  existing  training  evaluation 
system  in  terms  of  these  five  purposes  is  clearly  beyond  the 
scope  of  this  report.  Instead,  we  have  chosen  to  focus 
attention  on  the  issues  of  content  validity  and  training 
efficiency  as  they  form  the  initial  steps  for  answering 
questions  regarding  the  three  additional  evaluation 
purposes. 


Content  Validity  Issues 

One  of  the  purposes  of  an  evaluation  system  is  to 
insure  that  the  training  content  is  job  relevant  or  content 
valid,  ("is  the  training  content  job  relevant?") .  The 
information  needed  to  address  this  question  includes  an 
analysis  of  the  "task"  components  of  the  job  and  training 
content  domains.  Thus,  the  question  of  content  validity 
must  be  answered  through  a  direct  comparison  of  the  tasks 
(or  knowledges  and  skills)  being  performed  on  the  job  with 
the  tasks  being  taught  in  the  training  program.  The  greater 
the  overlap  between  the  tasks  performed  on  the  job  and  the 
tasks  taught  in  the  training  program,  the  greater  the 
content  validity  of  the  training.  Tasks  in  the  training 
program  that  are  not  important  on  the  job  are  considered 
contaminants,  while  important  tasks  not  included  in  the 
training  program  are  considered  training  deficiencies. 

Therefore,  to  determine  the  content  validity  of  a 
training  program,  the  content  of  the  job  in  terms  of  tasks 
performed  and  the  content  of  the  training  program  must  be 
identified.  Currently,  the  job  content  domain  is  specified 
at  the  task  level  by  the  OSR.  The  ABR  training  content 
domain  is  most  clearly  identified  at  the  task  cluster  level 
by  the  STS. 

Therefore,  it  is  difficult  to  determine  training 
content  validity  at  the  task  level  of  specificity.  To  more 
comprehensively  evaluate  content  validity,  the  content  of 
the  training  program  must  be  clearly  identified  at  the  task 
level.  Without  direct  information  about  the  training  content 
domain  at  the  task  level,  there  is  no  simple,  systematic  way 
of  incorporating  the  new  OSR  information  into  the  evaluation 
system  to  make  meaningful  changes  in  training  content  and 
training  design.  Instead,  instructor  knowledge  of  course 
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content  is  required  to  determine  if  an  OSR  task  is  currently 
taught  in  the  AGE  ABR  course.  Given  duty  rotations,  it  is 
highly  likely  that  the  instructors  who  helped  make  linkages 
between  the  old  OSR  and  the  POI  are  not  available  when  the 
new  OSR  data  is  created.  Thus,  linkage  information  may  be 
lost  to  the  evaluation  system. 

Training  Efficiency  Issues 

A  content  valid  training  program  indicates  that  the 
tasks  being  performed  on  the  job  are  the  same  tasks  being 
trained.  Once  content  validity  is  established,  a  second 
issue  is  training  efficiency.  Training  efficiency  asks  the 
question  "is  the  training  program  over-  or  undertraining 
certain  tasks  or  KSAs?."  Thus,  a  training  program  may  have  a 
high  degree  of  job  relevancy  (training  the  appropriate 
tasks)  but  may  or  may  not  be  placing  an  appropriate  amount 
of  emphasis  on  the  various  tasks  during  training  to  match 
their  "need"  for  training.  The  information  needed  to 
address  the  training  efficiency  question  includes  an 
analysis  of  the  "emphasis"  components  of  the  job  and  the 
training  content  domains  (see  Figure  1) .  The  question  of 
efficiency  can  be  addressed  through  a  direct  comparison  or 
matching  of  the  emphasis  placed  on  the  tasks  during  training 
with  training  "needs"  i.e.,  how  important  the  task  is  in  the 
job  domain. 

Ford  and  Wroten  (1984)  developed  a  methodology  called 
the  Matching  Technique  to  link  the  "emphasis"  components  of 
the  job  and  training  content  domains  to  determine  training 
efficiency.  The  Matching  Technique  is  conceptualized  in 
Figure  2  as  a  matrix  in  which  training  emphasis  is  directly 
compared  to  training  needs"  (i.e.,  how  important  the  task  is 
for  job  performance) . 

The  comparison  of  emphasis  with  needs  identifies 
training  "hits"  and  "misses."  Training  hits  refer  to  those 
tasks  where  the  emphasis  received  in  training  appropriately 
reflects  training  needs.  Training  misses  can  involve  areas 
of  deficiencies  (undertraining)  and  excesses  (overtraining) . 
Training  deficiencies  are  content  areas  whose  high  training 
needs  are  not  matched  by  a  high  degree  of  emphasis  in  the 
training  program.  Training  excesses  are  tasks  that  are 
receiving  an  excessive  amount  of  emphasis  relative  to  their 
need  to  be  training.  The  greater  the  number  of  misses  (both 
deficiencies  and  excesses)  uncovered,  the  less  efficient  the 
training  program  is  and  the  greater  need  for  reassessment 
and  redesign  of  the  program. 

Any  measure  of  training  "needs"  or  training  "emphasis" 
can  be  used  to  apply  the  Matching  Technique  as  long  as  the 
measures  reliably  quantify  the  extent  to  which  areas  need  to 
be  trained  or  emphasized  in  the  training  program. 
Nevertheless,  as  noted  when  discussing  content  validity,  the 
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POSSIBLE  TRAINING 
EXCESSES 


comparison  of  training  "needs"  (i.e.,  job  emphasis  or 
importance)  and  training  emphasis  can  only  be  as  good  as  the 
quality  of  the  data  collected.  Also  the  type  of 
information  (tasks,  KSAs)  collected  on  the  job  and  training 
content  domains  must  be  operationalized  at  the  same  level  of 
specificity  to  make  this  direct  comparison  possible. 

An  examination  of  the  evaluation  information  collected 
for  the  AGE  ABR  course  reveals  that  the  Matching  Technique 
can  not  be  completed  as  some  of  the  information  needed  is 
not  available.  The  training  emphasis  rating  found  in  the  OSR 
is  the  supervisor's  perception  of  which  OSR  tasks  should  be 
taught  in  the  AGE  ABR  course.  The  problem  is  that  the 
training  content  domain  has  no  comparable  training  emphasis 
information.  Instructor  ratings  of  the  emphasis  placed  on 
tasks  during  training  or  the  actual  time  spent  in  training 
on  tasks  are  two  ways  of  collecting  emphasis  information  in 
the  training  content  domain. 

Another  factor  limiting  the  ability  of  the  training 
system  to  adequately  address  the  training  efficiency  issue 
is  the  lack  of  adequate  training  and  job  performance 
information.  Although  performance  information  is  available 
in  both  the  job  and  training  domains,  the  information  is 
neither  used  nor  collected  in  a  form  that  makes  it  useful  to 
the  training  evaluation  system.  Performance  information 
would  be  quite  useful  in  determining  over-  or  undertraining 
of  tasks  in  the  AGE  ABR  course.  This  issue  is  addressed 
further  in  the  future  research  directions  section  below. 

V.  FUTURE  RESEARCH  DIRECTIONS 

Based  on  the  conceptual  models  developed  and  the 
analysis  of  the  Air  Force  ABR  Training  system,  two  research 
needs  are  identified:  (a)  to  systematically  identify  the 
training  content  domain  in  terms  of  tasks  and  emphasis 
placed  on  those  tasks,  and  (b)  to  incorporate  job 
performance  information  into  the  training  system  to  address 
the  issue  of  training  efficiency. 

Identification  of  Training  Content  Domain 

The  job  content  domain  is  defined  through  the  OSR 
process  in  terms  of  the  tasks  performed  on  the  job  and  the 
emphasis  that  should  be  placed  on  training  those  tasks 
during  the  ABR  course.  The  OSR  process  represents  a  large 
commitment  of  resources  to  develop  systematic  data  on  the 
job  domain  which  can  be  used  for  aiding  training  personnel 
of  the  ABR  courses  in  updating  training  to  maintain  a  high 
quality  program. 

A  major  problem  identified  in  this  review  is  that 
similar  efforts  to  systematically  identify  the  ‘-asks  taught 
and  the  emphasis  placed  on  those  tasks  in  the  training 
content  domain  are  lacking.  Information  about  the  ABR 
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course  is  available  through  the  POIs  and  lesson  guides  but 
is  not  at  the  same  level  of  specificity  as  the  OSR  data.  In 
addition,  no  data  are  gathered  on  how  much  emphasis  is 
actually  placed  on  each  task  that  is  part  of  the  training 
content  domain.  Without  such  information,  decision  makers 
can  not  link  the  job  and  training  content  domains  to  fully 
answer  questions  of  training  content  validity  and  training 
efficiency. 

Consequently,  one  research  effort  should  be  devoted  to 
developing  a  methodology  for  identifying  the  training 
content  domain  in  terms  of  tasks  and  emphasis.  Training 
Branch  personnel,  instructors,  students  and  recent  graduates 
are  all  potential  sources  of  information  regarding  the 
training  content  domain. 

For  example,  TDB  staff  members  from  AGE  can  be  used  as 
SMEs  and  asked  to  independently  examine  the  list  of  OSR  task 
statements  from  the  most  recent  OSR  (i.e.,  1983)  and  state 
whether  the  task  is  taught  or  not  in  the  ABR  course. 
Disagreements  among  SMEs  can  be  resolved  through  consensus 
judgment  to  provide  a  preliminary  list  of  OSR  task 
statements  that  are  taught  in  the  ABR  course.  Given  that 
the  ABR  course  is  divided  into  five  training  areas,  the  SMEs 
could  then  match  the  preliminary  list  of  task  statements  to 
the  area  in  which  each  task  is  taught  (some  tasks  may  be 
taught  across  a  number  of  areas) .  This  will  provide  a  task 
matrix  for  each  of  the  five  training  areas  in  the  course. 
Once  the  matrix  is  created,  instructors  can  then  be  used  to 
verify  the  task  list.  Multiple  instructors  are  assigned  to 
each  area  and  each  instructor  leads  a  class  of  graduates 
(usually  12)  through  his/her  training  area.  Thus,  surveys 
can  be  conducted  to  determine  whether  there  is  substantial 
agreement  among  the  various  instructors  that  the  tasks 
identified  are  actually  taught  in  the  training  program.  This 
two  step  process  (TDB  staff  SME  judgments  and  instructor 
survey  responses)  would  result  in  a  list  of  tasks  taught  by 
area  at  the  OSR  task  level  of  specificity.  It  would  also 
allow  for  an  examination  of  the  extent  to  which  instructors 
in  the  same  area  teach  the  same  tasks. 

Once  the  training  content  domain  has  been  identified  at 
the  OSR  level,  the  second  phase  of  the  research  would  be  to 
develop  a  methodology  for  collecting  information  on  how  much 
emphasis  is  actually  placed  on  each  task  trained  in  the  AGE 
ABR  course.  Emphasis  information  can  be  collected  from  two 
sources.  First,  information  regarding  the  amount  of  time 
actually  spent  training  each  task  identified  in  Phase  1  can 
be  determined  through  a  survey  of  the  AGE  instructors.  The 
survey  would  ask  each  instructor  to  independently  estimate 
the  amount  of  time  spent  training  each  task  in  his/her 
training  area  i.a.,  to  distribute  the  total  time  available 
for  training  in  an  area  across  the  tasks  being  taught  in 
that  area.  This  would  allow  for  an  analysis  of  the  extent 
to  which  different  instructors  in  the  same  training  area 
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place  the  same  amount  of  emphasis  on  different  tasks  trained 
in  that  area.  Second,  ratings  of  "relative  emphasis"  given 
to  each  task  trained  can  be  collected  through  a  survey  of 
recent  graduates  of  the  AGE  ABR  course.  This  would  allow 
for  a  comparison  of  the  instructor  estimates  of  training 
emphasis  with  estimates  of  recent  graduates. 

Integrating  Training  Efficiency  and  Performance  Information 

The  identification  of  the  training  content  domain  in 
terms  of  OSR  tasks  and  the  development  of  a  methodology  for 
determining  training  emphasis  are  prerequisites  for  using 
the  Matching  Technique  (Ford  &  Wroten,  1984) .  The  Matching 
Technique  provides  preliminary  information  that  certain 
tasks  may  be  training  excesses  (i.e.,  overtrained)  or 
training  deficiencies  (i.e.,  under Lrained) .  To  more  fully 
address  the  question  of  training  efficiency  (are  tasks  over- 
or  undertrained?) ,  performance  information  must  be 
incorporated  into  the  process. 

Figure  3  presents  a  2  X  2  matrix  which  combines 
information  that  is  created  from  the  Matching  Technique  with 
job  performance  information.  An  analysis  of  the  Matching 
Technique  identifies  tasks  that  are  excesses  and 
deficiencies.  In  Figure  3,  performance  level  is 
conceptualized  at  the  group  level  of  analysis  i.e.,  how  well 
the  group  of  individuals  trained  are  performing  various 
tasks  on  the  job.  For  purposes  of  this  example, 
performance  is  dichotomized  into  the  categories  of  "not 
performing  well"  and  "performing  well".  The  four 

components  of  the  matrix  in  Figure  3  derived  by  combining 
the  outcomes  of  the  Matching  Technique  with  performance 
level  information  provide  data  relevant  to  identifying  over- 
or  undertrained  tasks.  As  an  example,  let  us  assume  that  a 
number  of  tasks  were  identified  through  the  Matching 
Technique  as  being  deficient.  The  next  step  is  to  examine 
how  well  recent  graduates  are  actually  performing  those 
tasks  on  the  job.  If  the  tasks  are  not  being  performed 
well,  then  the  need  for  more  training  emphasis  during  the 
training  program  is  indicated.  If  the  tasks  are  being 
performed  well,  this  indicates  that  training  deficiency  is 
not  an  important  issue  to  consider.  Such  a  result  may 
suggest  that  formal  training  of  the  tasks  is  not  necessary. 

The  Matching  Technique  may  also  identify  some  tasks  as 
training  excesses.  An  examination  of  job  performance  may 
indicate  that  some  tasks  that  are  training  excesses  are 
performed  well  and  others  are  not  performed  well  (see  Figure 
3) .  If  a  task  is  not  performed  well  on  the  job,  this 
indicates  that  there  is  a  serious  problem  that  is  best 
addressed  through  other  means  such  as  on-the-job  training. 

If  a  task  is  performed  well,  this  suggests  that 
consideration  be  given  to  examining  whether  training 
emphasis  placed  on  the  task  could  be  reduced  without 
detrimental  effects  on  job  performance.  Currently,  job 
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figure  3.  Model  of  TralnlnQ  Efficiency  Information  and  Job  Performance  Information. 


performance  information  is  not  used  to  make  decisions 
regarding  training  efficiency.  A  new  methodology  called  the 
Job  Performance  Measurement  System  (JPMS)  developed  by  the 
AFHRL  (Hedge  &  Teachout,  1986)  provides  the  type  and  quality 
of  job  performance  data  needed  to  identify  over-  and 
undertrained  tasks. 

The  JPMS  contains  three  types  of  information.  First, 
participants  complete  "a  hands-on"  portion  of  the  Walk- 
Through  Performance  Testing  procedure  in  which  participants 
actually  perform  certain  tasks  and  are  scored  by  trained 
observers  on  whether  the  tasks  are  performed  correctly.  A 
second  portion  of  the  Walk-Through  Performance  Test  is  an 
interview  portion  which  asks  participants  to  explain  how 
they  would  go  about  completing  certain  tasks  rather  than 
actually  performing  those  tasks.  These  responses  are  scored 
by  trained  interviewers.  Both  the  hands  on  testing  and  the 
interview  method  are  scored  at  the  TO  (Step)  level  (subtasks 
are  performed  correctly  or  not)  which  are  then  summed  for 
each  task  to  provide  a  score  at  the  OSR  (task)  level.  The 
third  type  of  information  is  a  Specialty  Job  Knowledge  Test 
which  covers  a  number  of  tasks  in  the  career  field. 
Participants  must  respond  to  a  number  of  multiple  choice 
questions  derived  from  the  task  content  domain  at  the  task 
level  of  specificity. 

Given  the  development  of  the  conceptual  model,  the 
next  step  is  the  development  of  the  methodology  to  integrate 
JPM  data  into  the  training  evaluation  system  to  address 
issues  of  over-  or  undertraining.  If  the  training  content 
domain  is  specified  at  the  task  level  of  specificity,  then 
the  integration  of  performance  data  must  be  at  the  task 
level  also.  Nevertheless,  for  tasks  that  are  identified  as 
over-  or  undertrained,  job  performance  at  the  TOSL  can  then 
be  examined  to  determine  which  procedural  steps  are  most 
problematic.  Research  could  examine  the  impact  of  feeding 
back  job  performance  information  to  training  personnel  on 
ABR  course  changes. 
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Appendix  A:  SUPPORTING  AIR  FORCE  DOCUMENTS 


Specialty  Training  Standard 
Occupational  Survey  Report 
Training  Extract 
Training  Evaluation  Report 
Training  Evaluation  Report 
Training  Evaluation  Report 
AGE  Utilization  and 
Training  Workshop 
Study  Guide/Workbook 
Plan  of  Instruction 
Training  Plan 
Instructional  System 
Development 


423X5 

AFPT  90-423-459 
AFPT  90-423-459 
TER  C  85-5 
TER  C  86-28 
TER  C  87-22 

Minutes 

C3AIR75135 

C3ABR42335 

C3ABR42335 

AFM  50-2 


July  1982 
May  1983 
May  1983 
February  1986 
February  1987 
February  1338 

December  1983 
June  1987 
August  1987 
June  1987 

July  1986 
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APPENDIX  B 


department  of  the  AIR  force 

Headquarters,  USAF 
Washington  DC  20330-5000 


STS  423X5 
(For  AFSCs  42335/55/75) 
August  1987 


AEROSPACE  GROUND  EQUIPMENT  MECHANIC 
AND 

AEROSPACE  GROUND  EQUIPMENT  TECHNICIAN 

1.  The  Implementation  of  the  STS  for  technical  training  provided  by  Air  Training  Cociaand  Is  wltn 
.h«  crass  entering  8/1005  and  graduating  680209. 

2.  Purpose  of  this  Specialty  Training  Standard  (STS).  As  prescribed  in  AFR  8-13,  this  STS: 

a.  Lists  In  column  1  of  attachment  1  the  most  common  tasks,  knowledges,  and  technical 
references  (TR)  necessary  for  airmen  to  perform  dudes  at  the  3-,  5-,  and  7-sktll  level  AFSC  In  the 
Aerospace  Ground  Equipment  ladder  of  the  Airman  Aircraft  Accessory  Maintenance  career  field.  These 
are  based  on  the  analysis  of  the  ducles  in  AFR  39-1,  effective  31  October  1986. 

b.  Provides  OJT  certification  columns  In  attachment  1  to  record  completion  of  task  and 
knowledge  training  requirements.  Certification  Is  accomplished  as  outlined  In  AFR  50-23. 

c.  Shows  formal  training  requirements.  Column  3 A ( 1 )  of  attachment  l  shows  the  proficiency 
to  be  demonstrated  on  the  Job  by  the  graduate  as  a  result  of  training  in  Course  C3A8R42335  000 
(PDS  Code  ABU)  described  In  AFR  50-5.  When  two  codes  are  used  In  column  3A(1),  tne  first  code  Is 
the  established  requirement  for  resident  training  on  the  task/knowledge.  The  second  code  Is  the 
level  of  training  provided  by  the  course  due  to  resource  constraints.  Column  3C(1)  of  attachment  1 
shows  the  proficiency  to  be  demonstrated  on  the  Job  by  the  graduate  as  a  result  of  training  In 
Course  C3AAR42375  000  (PDS  Code  A8J ) . 

d.  Becomes  a  job  qualification  standard  for  on-the-job  training  when  placed  In  AF  Form  623, 
On-the-Job  Training  Record,  and  used  according  to  AFR  50-23.  For  OJT,  the  tasks  In  column  1  are 
trained  and  qualified  to  the  go/no  go  level.  Go  means  the  Individual  can  perform  the  task  without 
assistance  and  meet  local  requirements  for  accuracy,  timeliness,  and  correct  use  of  procedures. 

e.  Indicates  In  column  3B(2)  career  knowledge  provided  In  the  5-sklll  level  CDC.  See 
ECI/AFSC/CDC  listing  maintained  by  the  unit  OJT  manager  for  current  CDC  listings. 

f.  Is  a  guide  foe  development  of  promotion  testa  used  In  the  Weighted  Airmen  Promotion 
System  (WAPS).  Specialty  Knowledge  Tests  (SKTs)  are  developed  at  the  USAF  Occupational  Measurement 
Center  by  senior  NCOs  with  extensive  practical  experience  In  their  career  fields.  The  tests  sample 
knowledge  of  STS  subject  matter  areas  Judged  by  test  development  team  members  to  be  most 
appropriate  for  promotion  to  higher  grades.  Questions  are  based  on  study  references  listed  In 

AFP  39-8.  Individual  responsibilities  are  In  Chapter  14  of  AFR  35-8. 

g.  Attachment  2  provides  the  electronic  fundamental  requirements  for  this  specialty.  Only 
chose  items  coded  are  required  by  this  AFSC. 

3.  Proficiency  Code  Key.  Attachment  1  concalns  the  proficiency  code  key  used  to  Indicate  the 
level  of  training  and  knowledge  provided  by  resident  training  and  career  development  courses. 

4.  Recommendations ■  Report  unsatisfactory  performance  of  Individual  course  graduates  as 
prescribed  In  AFR  50-38.  Report  Inadequacies  of  this  STS  through  channels  to  HQ  ATC/TT0. 

Reference  specific  STS  paragraphs. 

BY  ORDER  OF  THE  SECRETARY  OF  THE  AIR  FORCE 


OFFICIAL  LARRY  D.  WELCH,  General,  USAF 

Chief  of  Staff 


NORMAND  G.  LEZY,  Colonel,  USAF 

Director  of  Information  '!.m  icront 
and  Adrr  mUtrat  ion 


2  Atch 

1.  Qualitative  Requirements 

2.  Electronic  Fundamentals/Appl  teat  tons 


Supersedes  STS  42JX5,  May  1984. 
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STS  423X5 


THIS  BLOCK  IS  FOR  IDENTIFICATION  PURPOSES  ONLY 

NAME  OF  TRAINEE 

PRINTfo  HAM*  (Loti,  FOof,  Middle  Initial)  INITIALS  (Written)  HAN 

PRINTED  NAME  OF  CERTIFYING  C 

IFF ICIAL  AND  WRITTEN  INITIALS 

N/1 

M/I 

N/l 

M/I 

M/I 

M/I 

M/I 

N/l 

N/l 

M/I 

M/I 

M/I 

QUALITATIVE  REQUIREMENTS 


PROFICIENCY  CODE  KEY 

Ml 

SCALE 

VALUE 

DEFINITION:  The  individual 

TASK 

PERFORMANCE 

LEVELS 

b 

Can  do  simple  parts  of  tha  task.  Naads  to  ha  told  or  shown  how  to  do  most  of  tha  task. 

(EXTREMELY  LIMITED) 

■1 

B 

Cm  da  all  carta  of  the  laak.  Naoda  only  a  tool  ckack  of  completed  work.  (COMPETENT) 

iq 

Can  do  tha  complete  taak  quickly  and  accurately.  Can  toll  or  shew  o then  now  lo  do  the  teak. 

(HIGHLY  PROFICIENT) 

Ul 

2»> 

X 

B 

Con  aama  oarta,  tool*,  and  elmpta  foot*  about  tha  taak.  (NOMENCLATURE) 

Can  oa  tormina  atae  By  ft  op  proooouroa  for  dolnq  the  taak.  (PROCEDURES) 

B 

Can  eredlat,  Isolate,  and  roaol*e  proBMmi  about  tha  Uak.  (ADVANCED  THEORY) 

••  SUBJECT 
KNOWLEDGE 
LEVELS 

B 

Can  Identify  baalc  facta  and  terma  about  tha  subject.  (PACTS) 

B 

Can  Identify  relationship  of  basic  facta  and  itata  tanaral  principles  about  the  subject.  (PRINCIPLES) 

c 

B 

Can  evaluate  concitians  and  make  prayer  dccfalons  about  the  subject.  (EVALUATION) 

EXPLANATIONS 

*  A  Ink  know1*di«  Mii«  *mhi«  m mt  be  oaad  ilooi  or  with  a  Uak  performance  tali  nlu«  to  define  «  Uwl  of  tao«IM|«  lot 
•  apBdfletMk.  (InapUi:  band  lb) 

••  A  NbM  1bowI*4m  *Mh  Mhi«  la  uwd  tJooa  to  daftaa  a  levs)  of  bovl*4f«  for  a  wbtact  not  SmO?  rata  tad  to  any  iptdfli 
task,  or  for  a  mM  common  to  aaaaral  u«kj. 

Thke  mark  It  need  aloot  hittaad  of  a  teal*  mhia  to  Miow  that  no  proficiency  training  la  provided  In  th«  count  or  COC. 

X  Tkli  mark  It  oaad  atone  ha  to  *nt  cohimaa  to  dot  that  training  u  required  but  not  given  dut  to  UnUlattont  In  ratouratt. 
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CEHTIf  (CATION  fO «  QJT 


TASKS.  KNOWLEDGE  *KO 
TECHNICAL  «€FEP€NCES 


Ck«w**ktl#« 

am 


C«rtifyl»« 


Nut e  l :  Lfd  *  r i 
cevl  s  Lon. 


responsible  for  annotating  cralnlng  referenced  to  Identify  current  references  pendlhg  STS 


J.  FHOFICIENCY  COOES  USED  TO  iNOI 
CATC  TRAlNiNC/lNFORMATiON  PROVIDE O 


A  • 

)  Skill  Lt*M  I  *  Skill  l«««* 


( * i  i2i  j  in  m  m  in 

Comii  I  COC  C«*n*  COC  Clviw'  C  CX 


Note  2:  ‘Tactical  -Vic  Control  Systems  (TACS ) /Ground  L lunched  Cruise  Miss  .le  (CLCM;  electron  .c  Generating 
Equipment  tcalnLng  in  STS  »ilements  24(a)  tnreugh  24(f)  is  accomplished  In  Courses  < 3AZR42355  001  and 
C3A2R42J55  o02  as  supplemental  training  for  personnel  isslgned  to  TACS/GLOM  units.  Thl$  supplemental 
training  does  not  apply  to  normal  AKSC  resident  tralnlig  (C3A8JU2335),  C*  eer  Development  Course  (COC),  ir 
Specialty  Knowledge  Tests  ( 5KTs ) .  However,  taslc  TAC3  orientation  and  advanced  electronic  principles  ire 
tnco rpor  iced  In  tne  COC,  imi  may  be  Included  In  the  SK’s. 

Note  3:  It-ms  marked  with  an  asterisk.  (*)  ace  tne  era  .nLng  standard  elements  thee  are  supported  when 
determining  the  uact Ime  e  >ucse  lengths  In  resident  couses. 

: 

1.  AEROSPACE  ttOUNO  EOl'IPME.T  CAREER  1 

LADOER  PROCRESS  r-jh  B 

IR:  AFR  39-1 

2.  OPERATION*  SECl'-tLCt  )PSEC*  SPECIFIC 

OPSEC  VULNERABILITIES  OF  \FSC  423X5  A 

TR:  AFR  55-30 

3.  AIR  FORCE  OCCUPATIONAL  SAFETY  ANO 
HEALTH  ( AFOSH  )  PROGRAM 

TR:  AfRs  1  27  -2,  1  2  7-1  2  ;  .Applicable  1 

OSHA  and  AFOSH  standards  ^ 

a.  Supervisory  responsibilities  -  I  C 

*b.  Individual  responsibilities  B  I  ,  -  C 

*c.  Practice  job  satety 

*d.  Fire  Prevention 

*e.  Occupational  Health 

4.  TECHNICAL  PUBLICATIONS 

♦a.  Fundamentals  ot  TO  System 
TR;  TO  JO- 5- 1 

*b .  Use  fnde*  Type  Teen  Orders 

TR:  TOs  0-l-0:.  0-2-1, 

vO-5-1,  JU-5-2 

c.  Use  Tech  Manuals  is  source  of 
Information  tor  performing: 

TR:  TO  00-5-1,  TOs  for  specific 
equipment  ind  ippllcible 
abbreviated  tech  orders 

*  ( 1  )  Maintenance 


*( 2)  Inspect  tons 
Use  methods  and  procedures 
Technical  Orders 
TR:  TO  00-xx  Series 


(l)  Maintenance  information 


(2)  Management  Infjrmstljn 


J 


Attachment  l 


NO  CDC  REQUIREMENT 


STS  423X5 


tasks.  *no*ledce  and 
TECHNICAL  references 


TECHNICAL  PUBLICATIONS  iconttnued) 

,3.'  Adaluistrat  !ve  Utoriatloo 
e.  Use  Standard  Publication*  to 
determine  tfte  following 
Information  pertinent  to 
ma  mtenance 

TR  :  AFRs  O-xx  Series.  Subject 
Series  50,  52,  66,  122,  and 
applicable  OSllA  and  AFOSH 
Standards 


1 1 ) 

Policies 

(2) 

Procedures 

(3) 

Inst  rue  t  Ions 

f.  Initiate  TO  Improvement  Report 
*g.  Comply  with  TCTOs 
5.  SUPERVISION  AND  TRAINING 

a.  Supervision 

(1)  Obtain  Information  for: 

TR:  AFM  67-1  (vol  II, 
part  1,  chaps  5  and  7); 

AFRs  66-1,  67-23  (chaps  3 
and  4);  TO  00-20-3  (sec  II) 

(a)  Special  requisitions 

(b)  Issue  slips 

(c)  Turn-In  slip* 

(2)  Prepare  equipment 
authorization  list 

TR:  AFM  67-1  (vol  IV, 
part  1,  chap  7);  AFRs  66-1, 
67-23,  (chap  3,  sec  B) 

(3)  Statement  of  charges 

TR:  AFN  67-1  (vol  IV, 

part  1 ) 

(4)  Report  of  Survey 

TR:  AFR  177-11; 

TO  00-350-780  (Sec  III) 

(5)  Coordinate  work  with  other 
personnel 


TR: 

AFRs 

39-6,  66-1 

Plan 

TR: 

AFRs 

39-6,  66-1 

(a) 

Work 

ass Ignment  s 

(b) 

Work 

priori  ties 

SIAM 


CERTIFICATION  for  ojt 


Oat* 


Cim'r'M 

OMitlSl'l 

IWIItlt 


T /  Imff  l 
IRilllll 


3  PROFICIENCY  COOES  USE  D  TO  *NO« 

CATe  TWainiNC/iNF QRMA TlQN  ^RQVIOEO 


Ciuiit  COC 


%  **••>  L«e«l 


t 


i* i ij»  ]  ( ti *  <*• 
C*m»,  COC  C  1  COC 


b 

b 

!  b 
b 
!  b 
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